Redundant communication fabrics for enhancing fault tolerance in Totem networks

ABSTRACT

Disclosed is a method and apparatus for providing fault tolerance in Totem Networks by use of redundant fabrics. The above is accomplished in one embodiment of the invention by operating devices on the network in such a way that the devices mark the token to indicate when the token has been switched from one fabric to another in response to a timeout. A Ring Master device on the network determines, based on switching of the token by devices on the network whether a fabric or device on a fabric of the network has failed. In addition, fabrics that have failed are monitored to determine when they have become operational. Retransmission of improperly received messages as per token-message-order protocols are also provided for situations in which the token is received before all messages intended for a given device have been properly received.

TECHNICAL FIELD

The present invention relates in general to communication systems and,more particularly, to the use of redundant communication fabrics toenhance fault tolerance in Totem communication networks.

BACKGROUND OF THE INVENTION

A number of systems have been developed for providing networkcommunications among groups of users. One such system comprises a Totemring network in which a plurality of devices is connected to a busnetwork. Each communication device includes circuitry for interfacingwith the Totem ring network (e.g., transmitting and receiving messageson the Totem ring network), and a Central Processing Unit (CPU) adaptedfor executing processes comprising application programs effective formanaging call processing, database operations, industrial control, andthe like.

A Totem network provides for multicast delivery of messages, whereinmessages can be transmitted and delivered to multiple locations, withassurance that the sequence in which messages are generated ismaintained as the messages are transmitted and delivered throughout thesystem. Totem networks are well known to those skilled in the art andare described in greater detail in various technical papers andarticles, such as an article entitled “Totem: A Fault Tolerant MulticastGroup Communication System” by L. E. Moser et al., published in theApril 1996, Vol. 39, No. 4 Edition of Communications of the Associationfor Computing Machinery (ACM).

In Totem networks, message delivery is controlled using a token similarto that used in a token ring system to identify which device cantransmit onto the network. Periodically, such as every few milliseconds,the token is sent around the network to each device in sequence. As thetoken is received by each device, the device determines whether it has amessage or data to transmit over the network. If a device does have amessage or data to transmit over the network, it will send that datafirst before forwarding the token. If a device does not have a messageor data to transmit over the network, then it forwards the token andsends it to the next device.

Conventionally, messages on a Totem network are transmitted anddelivered over a physical medium comprising a single fabric of wires orfiber optic cable. As a consequence, while Totem networks assure thatmessages are transmitted and delivered in the same sequence in whichthey are generated, there is no assurance that the messages will bedelivered at all if a fabric fails. The physical medium of a Totemnetwork thus has no fault tolerance designed into it.

Accordingly, there is a need for a system and a method that will provideTotem networks with fault tolerance to enhance the probability thatsequentially transmitted messages will be delivered across the Totemnetwork.

SUMMARY OF THE INVENTION

The present invention accordingly provides a Totem network with multipleredundant fabrics through which messages can be transmitted anddelivered. The Totem network is configured so that, if one fabric fails,another fabric can be used, thereby providing a Totem system with faulttolerance. The Totem network is also configured so that if a failedfabric has been repaired and thus becomes operational, the fabric repaircan be detected and the repaired fabric declared operational so thatdevices on the network can use it. The Totem network is also configuredso that a failure of a device on the network can be detected.

The present invention further comprises a method embodied in computersoftware residing on the network for controlling the use of theredundant fabrics. The computer software can be configured to detectwhen a fabric failure has occurred, and, after a failure has beendetected, to declare the fabric to have failed so that devices on thenetwork will use only fabrics that are operational. In the event afailed fabric has been repaired, the computer software can detect therepair and declare the formerly-failed fabric operational so thatdevices on the network can use it. The computer software can also beconfigured to detect when a device on the network has failed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic diagram of a Totem ring network embodying featuresof the present invention;

FIG. 2 depicts a high-level conceptual diagram of a token used in thenetwork of FIG. 1;

FIG. 3 is a flow chart illustrating control logic for marking a tokenused in connection with a ring master device connected to the network ofFIG. 1 to indicate that a fabric of the network has failed;

FIG. 4 is a flow chart illustrating control logic for marking a tokenused in connection with a ring master device connected to the network ofFIG. 1 to indicate that a fabric previously determined to have failedhas become operational; and

FIG. 5 comprises a flow chart illustrating control logic for switchingfabrics by a communication device connected to the network of FIG. 1.

DETAILED DESCRIPTION

In the following discussion, numerous specific details are set forth toprovide a thorough understanding of the present invention. However, itwill be obvious to those skilled in the art that the present inventioncan be practiced without such specific details. In other instances,well-known elements have been illustrated in block diagram or schematicform in order not to obscure the present invention in unnecessarydetail. Additionally, for the most part, details concerning theoperation of Totem ring networks and the like have been omitted inasmuchas such details are not necessary to obtain a complete understanding ofthe present invention and are within the skills of persons of ordinaryskill in the relevant art.

Referring now to FIG. 1 of the drawings, the reference numeral 100generally designates a Totem network embodying features of the presentinvention. The Totem network 100 comprises a plurality of fabrics 101,two of which fabrics 102 and 104 are represented by solid-line ellipsesin FIG. 1, it being understood that the network 100 may comprise anynumber of fabrics greater than or equal to two, as indicated by themultiple dashed-line ellipses of FIG. 1. Each fabric 102 and 104comprises a physical medium well-known in the art, such as copper wires,fiber optic cables, or the like, and may be configured to operate usinga protocol such as 10 baseT or the like.

A plurality of communication devices well-known in the art, three ofwhich devices 114, 116, and 118 are depicted in FIG. 1, are eachoperably connected to each of the fabrics 102 and 104. At least one ofthe devices 114, 116, and 118, taken herein as the device 116, isarbitrarily chosen to serve as a ring master device of the Totemnetwork. Any of the devices may act as the ring master; however, if thering master fails, another device is chosen to serve as the ring master.The ring master device 116 manages tokens and messages to determine,based on fabric switching by devices 114, 116, and 118, whether afailure of any of the plurality of fabrics has occurred. The ring masterdevice 116 also contains a local fabric switch count register 120 foreach fabric that holds the number of consecutive fabric switches for thefabric.

The devices 114, 116, and 118 may comprise any conventional computergenerally capable of receiving, storing, processing, and outputtingdata. Each of the plurality of devices 114, 116, and 118 is configuredto switch transmission of a token among the plurality of fabrics inresponse to a timeout after transmission of a token. While not shown indetail, the devices 114, 116, and 118 include components, such as inputand output devices, volatile and non-volatile memory, and the like, but,because such computer components are well known in the art, they are notshown or described in further detail herein.

In FIG. 2, the reference numeral 200 generally designates a tokencomprising a plurality of data fields, three of which fields 202, 204and 206 are shown in FIG. 2, it being understood that the token 200 maycomprise any number of data fields. The data field 202 comprises normaltoken data used in tokens on prior art Totem ring networks, includingdata identifying which device 114, 116, or 118 the token 200 is intendedfor, which token data is well known in the art. The data field 204comprises information denoting which fabrics of the network 100 havefailed, as determined by the ring master device 116. The data field 206comprises information denoting the number of times that a device 114,116 or 118 of the network 100 has switched from one of the fabrics 102and 104 in response to a timeout following transmission of a token 200.

FIGS. 3-5 are flowcharts of control logic implemented by the devices114, 116, and 118, for managing the plurality of fabrics 102 and 104 inaccordance with the present invention.

FIG. 3 is a flow chart of control logic that can be implemented on thering master device 116 to operate as a failed-fabric detector inaccordance with the present invention. The control logic will beexemplified by showing how a fabric failure is detected by the ringmaster device 116, resulting in the marking of the token 200 to indicatethat a fabric 102 or 104 has failed.

In step 302, the ring master device 116 receives the token 200 from thedevice 118 on fabric 102. Execution then proceeds to step 304. In step304, the ring master device 116 determines whether the token 200 isintended for the ring master device 116. If the ring master device 116determines that the token 200 is intended for the ring master device116, execution proceeds to step 308. If the ring master device 116determines that the token 200 is not intended for the ring master device116, execution proceeds to step 306 and terminates.

In step 308, a determination is made whether a token fabric switchcount, stored in the field 206 of the token 200, is equal to zero. Thetoken fabric switch count 206 is incremented when a timeout occurs aftera device 114, 116, or 118 has transmitted the token 200 on one of thefabrics 102 or 104 and has not received the token 200 within apredetermined amount of time thereafter, such as within one millisecond.The token fabric switch count 206 is set to zero at step 320 by the ringmaster device 116 every rotation of the token 200 on the network 100.Thus, the token fabric switch count 206 stores the number of times thata fabric switch for a particular fabric 102 or 104 has occurred during atoken 200 rotation around the network 100.

If, in step 308, the token fabric switch count 206 is equal to zero,execution proceeds to step 309. If the token fabric switch count 206 isnot equal to zero, execution proceeds to step 312.

In step 309, the local fabric switch count 120 for the fabric the tokenwas originally sent on is set to zero because the token made asuccessful pass through the fabric without any retransmissions.

In step 312, a determination is made whether the total local fabricswitch count 120 for the fabric 102 exceeds a predetermined number, suchas 3. Other algorithms may be used for detecting poorly performingfabrics, such as counting the number of tokens which have been droppedduring an immediately preceding few seconds. The total local fabricswitch count 120 is stored by the ring master device 116, and reflectsthe number of token 200 rotations during which a token 200 transmittedby the ring master 116 on the fabric 102 has been switched by anotherdevice 114 or 118 to the fabric 104.

If, in step 312, the predetermined number of switches has not occurred,execution proceeds to step 314. In step 314, the total local fabricswitch count 120 for fabric 102 is incremented, and execution thenproceeds to step 310. In step 310, the token 200 is processed in awell-known manner in accordance with conventional Totem Ring networktechnology.

If, in step 312, more than a predetermined number of switches haveoccurred for the fabric 102, execution proceeds to step 316. In step316, the token 200 is marked to indicate that fabric 102 has failed.

From step 316, execution proceeds to step 318. In step 318, the localtoken fabric switch count 206 is set to zero. Execution then proceeds tostep 310, discussed above. From step 310, execution proceeds to step320, wherein the token fabric switch count 206 is set to zero. From step320, execution proceeds to step 322, wherein the token is transmitted onthe next non-failed fabric. For example, if the token 200 had beentransmitted on the fabric 102, in step 322, the token 200 may next betransmitted on the fabric 104. Upon completion of step 322, executionproceeds to step 324 and is terminated.

FIG. 4 is a flow chart of control logic that can be implemented on thering master device 116 to permit it to operate as a detector of a fabric102 or 104 that has failed and has subsequently become operational. Thecontrol logic will be exemplified by showing how a formerly-failedfabric that has now become operational is detected by a ring masterdevice 116, resulting in the marking of a token 200 to indicate that theformerly-failed fabric 102 or 104 is now operational.

Referring to FIG. 4, execution is initiated at step 401 and proceeds tostep 402 wherein a determination is made whether the token 200 wasmarked in step 316 to indicate that the fabric 102 or 104 has failed. Ifit is determined that a fabric 102 or 104 has failed, then executionproceeds to step 406; otherwise, execution terminates at step 404.

In step 406, the ring master transmits a test message on the fabric 102or 104 on which a failure has been detected. The test message will betransmitted around the fabric in the same order as a normal token by thedevices on the network. The ring master will receive and retransmit thetest message and count the number of times it has done this. Executionproceeds to step 407 where the ring master waits for the test message togo around the fabric some preferable large (i.e. 100) number of times.Execution then proceeds to step 408, wherein a determination is madewhether a timeout has occurred (i.e., whether the test message did notgo around the fabric in time), a timeout being defined as the expirationof a predetermined time period that would normally allow the testmessage to go around the ring the number of times required without thering master device 116 having received a response to the test message onthe failed fabric 102 or 104. If the predetermined time period has notbeen exceeded, execution returns to step 406. If, in step 408, it isdetermined that the predetermined time period has not been exceeded,execution continues to step 410. In step 410, the token 200 is marked toindicate that the fabric 102 or 104 that had failed is now operationaland available for use by devices 114, 116, and 118. Upon completion ofstep 410, execution terminates at step 412.

FIG. 5 is a flow chart of control logic that may be implemented ondevices 114 and 118 to permit them to operate as a fabric switch inaccordance with the present invention. The control logic will beexemplified by showing how the fabric 102 or 104 on which a token 200 istransmitted may be switched by a device 114 or 118 and a token fabricswitch count 206 incremented in response to detection of a timeout.

Referring to FIG. 5, execution is initiated in step 501 and proceeds tostep 502, wherein a token 200 is sent by a device 114 or 118 on a fabric102 or 104. Execution then proceeds to step 504.

In step 504, a determination is made whether a timeout has occurred, atimeout occurring when a predetermined amount of time (a timeout value)has elapsed before device 114 or 118 has received a token 200. Suchtimeout value is set to the worst-case time it would take the token togo around the ring under normal operation. If it is determined that atimeout has not occurred, execution terminates at step 506. If, in step504, it is determined that a timeout has occurred, execution proceeds tostep 508. In step 508, a token fabric switch count 206 is incremented.The token fabric switch count counts the number of times thattransmission of the token 200 has been switched from one of the fabrics102 or 104 to another of the fabrics 102 or 104. Execution then proceedsto step 510.

In step 510, the device 114 or 118 switches to another non-failed fabric102 or 104 for transmission of the token, depending on which fabric 102or 104 the token was received by device 114 or 118 on. Execution thenreturns to step 502.

By the practice of the present invention, fault tolerance of Totem ringnetworks is provided, which enhances the probability thatsequentially-transmitted messages will be properly delivered across the-Totem ring network. Because there are multiple redundant fabrics onwhich tokens and messages may be transmitted, in the event one or moreof the fabrics fails, tokens and messages can still be transmitted onthe network. Because the present invention also provides for detectionof the repair of a failed fabric, once a formerly-failed fabric becomesoperational, the network is alerted that the fabric is now operationaland devices on the network are able to use the fabric, thus resulting inincreased bandwidth and fault tolerance of the network.

It is understood that the present invention can take many forms andembodiments. Accordingly, several variations may be made in theforegoing without departing from the spirit or the scope of theinvention. For example, any number of fabrics, devices, and ring masterdevices can be used, so long as multiple or redundant fabrics areutilized to provide redundancy in the totem network.

Other methods may be employed to determine that a specific fabric hasfailed. For example, the number of times a token switch has occurred ona specific fabric over a period of time may be counted, or a device atwhich failures occurred may be recorded, to more accurately identifypoorly performing fabrics and to report the location of failure moreaccurately.

Other fabric recovery mechanisms may also be employed. For example, aresponse may be individually requested from each device in the network.

For improved performance in the event of a failure, tokens and messagesmay be sent on multiple fabrics, or on all fabrics, simultaneously sothat if a token is lost on one fabric it may be received on anotherfabric.

Having thus described the present invention by reference to certain ofits preferred embodiments, it is noted that the embodiments disclosedare illustrative rather than limiting in nature and that a wide range ofvariations, modifications, changes, and substitutions are contemplatedin the foregoing disclosure and, in some instances, some features of thepresent invention may be employed without a corresponding use of theother features. Many such variations and modifications may be consideredobvious and desirable by those skilled in the art based upon a review ofthe foregoing description of preferred embodiments. Accordingly, it isappropriate that the appended claims be construed broadly and in amanner consistent with the scope of the invention.

What is claimed is:
 1. A method for providing fault tolerance in a Totemnetwork, comprising the steps performed by a device operably connectedon the network of: receiving a token transmitted on a first fabric of aplurality of fabrics of the network; determining whether the number oftimes that a token has been switched from the first fabric to a secondfabric of the plurality of fabrics exceeds a predetermined number; upona determination that the number of times that a token has been switchedexceeds a predetermined number, marking the token to indicate that atleast one of the plurality of fabrics has failed; and setting the numberof fabric switches stored on the token to zero in response to theindication that at least one of the fabrics has failed.
 2. A method forproviding fault tolerance in a Totem network, comprising the stepsperformed by a device operably connected on the network of: receiving atoken transmitted on a first fabric of a plurality of fabrics of thenetwork; determining whether the number of times that a token has beenswitched from the first fabric to a second fabric of the plurality offabrics exceeds a predetermined number; upon a determination that thenumber of times that a token has been switched exceeds a predeterminednumber, marking the token to indicate that at least one of the pluralityof fabrics has failed; setting a fabric switch count stored on the tokento zero in response to the indication that at least one of the fabricshas failed; and transmitting the token on a second fabric of theplurality of fabrics of the network.
 3. A Totem ring master devicecomprising: a processor for processing messages and tokens; at least onefirst interface connectable to each fabric of a plurality of fabricscomprising a Totem ring network; means for determining whether thenumber of times the token has been switched from a first fabric to asecond fabric of the plurality of fabrics exceeds a predeterminednumber, thereby indicating that at least one of the plurality of fabricshas failed; and means for setting the number of fabric switches storedon the token to zero in response to the indication that at least one ofthe fabrics has failed.
 4. A Totem ring master device comprising: aprocessor for processing messages and tokens; at least one firstinterface connectable to each fabric of a plurality of fabricscomprising a Totem ring network; means for determining whether thenumber of times the token has been switched from a first fabric to asecond fabric of the plurality of fabrics exceeds a predeterminednumber, thereby indicating that at least one of the plurality of fabricshas failed; means for setting a fabric switch count stored on the tokento zero in response to the indication that at least one of the fabricshas failed; and means for transmitting the token on a second fabric ofthe plurality of fabrics of the network.